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WHAT SHOULD WE REALLY 
LEARN FROM NEW ORLEANS 
AFTER THE STORM? 


In July of 2018, the Education Research 
Alliance for New Orleans released a 
comprehensive, summative longitudinal 
report on the effects on student outcomes 
of the package of reforms implemented in 
New Orleans following hurricane Katrina in 
the fall of 2005. The following policy brief 
reviews the findings of this recent report 
by Douglas Harris and Matthew Larsen, 
offers critique of their methods and 
interpretation of findings and attempts to 
provide broader policy context for those 
findings. 


In summary, Harris and Larsen find 
Significant positive effects of Post-Katrina 
New Orleans school reforms on short-term 
student achievement measures, and longer 
term college attendance, persistence 

and completion. They attribute these 
results to the “market-based” reforms 
adopted following Katrina, and go to great 
lengths to dismiss or downplay threats 

to the validity of this conclusion. But for 
many reasons, that attribution may be 
misguided. 


* First, the authors downplay the 
potential influence of significant 
changes in the concentration of 
poverty across neighborhoods and 


schools—specifically the reductions 
in extreme poverty which may 
contribute significantly to the 
improved student outcomes in the 
years following Katrina; 


Second, the authors understate 

the importance of the substantial 
increases to funding which occurred 
concurrently with organizational 
and governance changes in the 
district, specifically disclaiming the 
importance of increased funding by 
Suggesting that the funding increases 
would not have existed but for the 
reforms; 


Third, the authors argue, without 
evidence, that similar funding 
increases provided to the old, New 
Orleans school system would not 
likely have had similar impact, 
claiming they would have been 
inefficient or wasteful. At the same 
time the authors sidestep the fact 
that much of the funding increase in 
the new system was allocated toward 
increased and duplicative overhead 
expenses, as well as increased 
transportation costs resulting from 
citywide choice; 


Fourth, the authors define the 
treatment as the package of market- 
based reforms, which are largely 
changes to the governance and 


organization of New Orleans schools, 
rather than focusing on the types of 
schools, programs and services, and 
qualifications of incoming staff who 
entered this marketplace. 


Adopting similar governance and 
organizational changes, and citywide 
choice in other contexts may lead 

to very different results. It remains 
unclear whether population change and 
redistribution, coupled with the infusion 
of resources could have resulted in similar 
effects, even without structural reforms. 
Introduction 


INTRODUCTION 


In July of 2018, the Education Research 
Alliance for New Orleans released a 
comprehensive, summative longitudinal 
report on the effects on student outcomes 
of the package of reforms implemented in 
New Orleans following hurricane Katrina 

in the fall of 2005. The following policy 
brief reviews the findings of that recent 
report by Douglas Harris and Matthew 
Larsen, offers critique of their methods 
and interpretation of findings and attempts 
to provide broader policy context for those 
findings. 


In brief, the authors find significant positive 
effects of Post-Katrina education reforms 
on both short run and medium-term 
student outcomes, including standardized 


assessment scores, graduation rates, 
college attendance, persistence and 
completion. The authors attribute the 
majority of these gains to market-based 
organizational reforms, dismissing the 
substantial influx of financial resources to 
New Orleans schools that accompanied 
those reforms, and downplaying the 
potential role of demographic changes 
over the studied period. The authors argue 
that they have accounted adequately for 
those changes in their models. In a related 
2015 op-ed based on earlier findings, 
Doug Harris, the principal investigator 

on the project, offers cautions regarding 
the extent to which New Orleans (NOLA) 
findings provide guidance for similar 
reforms elsewhere. 


As | will explain in this policy brief, the 
authors’ dismissal of the role of increased 
financial resources is wholly unsatisfactory 
as well as illogical. In fact, the authors 
specifically note that estimates from 

recent rigorous research “suggest that 

the increased spending could explain 

a substantial share of our estimated 
effects.” (p. 41) The authors downplay 

the causal effect of increased resources 

by suggesting that the reforms led to the 
increased resources and not vice versa. 
This explanation, however, intentionally 
sidesteps whether similar gains might have 
been made with increased resources alone, 
absent the structural and organizational 
reforms. 


The authors’ assertion that their data, 
methods and models sufficiently 

account for demographic changes is also 
problematic. The authors’ assert that while 
there may have been population shifts 
before and after the storm, the gains 
realized after the storm can be attributed 
to the reforms because the models account 
for those demographic shifts, and any 
differences which were not accounted for 
are likely trivial. 


| explain in this brief why binary 
indicators of falling above or below the 
185% income threshold for poverty are 
wholly insufficient for differentiating the 
conditions under which children live and 
attend school, especially in settings with 
very high child poverty rates and where 
poverty is geographically concentrated by 
neighborhoods. More precise measures 
than free or reduced priced lunch alone 
are required to fully account for changes to 
the level and clustering of poverty in New 
Orleans. 


A 2015 report from the Brookings 
Institution found that concentrated poverty 
declined more in New Orleans than in 
nearly every other city in the country 
between 2000 and 2013 (dropping from 
2nd to 40th in concentrated poverty among 
major U.S. cities).1 That decline must be 
considered as a relevant factor. 


The following brief explores these issues in 


greater depth. | begin by summarizing the 
major findings of Harris and Larson’s most 
recent technical report on New Orleans 
reforms. Next, | provide an overview of 
how policymakers and the media should 
evaluate this, and other similar research 

in terms of providing guidance for 
interventions which might be introduced 
or scaled up elsewhere. Policymakers must 
be able to identify the specific elements of 
any given reform package - or “treatments” 
studied - and how those reform elements 
interact with their policy context. NOLA 
“reforms” are particularly ill-defined from a 
research “treatment” perspective and thus, 
for deriving policy implications, and the 
NOLA policy context is especially unique. 


Next, | address the authors’ suggestion 
that the substantial spending infusion 
which accompanied reforms should not 
be considered a significant cause of the 
improved outcomes (or a significant 
element of the studied treatment), and 
the authors’ suggestion that demographic 
shifts and changes to child poverty 
concentration by neighborhood are likely 
inconsequential to their conclusions 
regarding the effectiveness of the 
“reforms.” Additionally, | point out that 
the reforms may have led to less efficient 
expenditure of the increased spending 
than might have occurred if spending was 
increased without the reforms. 


SUMMARIZING HARRIS & 
LARSON'S MAJOR FINDINGS 


Harris and Larsen summarize their major 
findings from their July, 2018 technical 
report as follows: 


We find that the package of reforms 
improved the quantity, quality, and equity 
of schooling in the city on almost every 
available measure, increasing average test 
scores by 0.28-0.40 standard deviations, 
high school graduation by 3-9 percentage 
points, college attendance by 8-15 
percentage points, college persistence 

by 4-7 percentage points, and college 
graduation by 3-5 percentage points. These 
effects translate to 10-67 percent increases 
over baseline levels. 


The reforms also apparently reduced 
educational inequality by race and income 
on most measures. Our estimation 
procedures address potential threats 

to identification, including, for example, 
changes in the population. The reforms 
highlight the potential of market-based 
school reforms, though we also identify 
reasons why effects of this large size 
and range may not be expected in other 
locations and circumstances. 


“Threats to identification” are factors 
other than the package of NOLA reforms 
that might explain why there was an 
improvement in student outcomes 


post-Katrina. Research that attempts to 
attribute the improvements to the reforms 
must account for these threats to be 
credible. 


Notable in this description is the authors 
insistence on referring to the “treatment” 
under investigation as the package of 
reforms or “market-based school reforms” 
which are, by the authors’ interpretation 
the “cause” of the changes in measured 
outcomes they observe via their selected 
empirical measures, methods and 

models. They observe (or estimate) these 
changes in various outcome measures 
from prior to, through the years following 
Katrina. They attribute these changes to 
the “package of reforms” implemented 
following Katrina, and put significant effort 
into dismissing other possible causes. 


There are indeed sizeable and important 
shifts in both short and longer term 
outcomes for students attending schools 
within the city of New Orleans following 
Katrina. Student outcomes are improved 
after the storm. Whether the authors’ 
attribution of the majority of these effects 
to “the package of reforms” is valid, and 
should be used to guide policy elsewhere, 
however, remains suspect. 


Doug Harris himself offers cautions 

regarding the interpretation of findings 
from earlier (2015) NOLA research.? For 
example, Harris explains that “...though 


disadvantaged students benefited, they 
seem to have benefited less than other 
groups.” Harris acknowledges concerns 
over services provided to children with 
special needs and the potential for 
disadvantaged students to fall through the 
cracks in a market-based, autonomous 
system. Harris also acknowledges that 

his early studies may be critiqued for too 
heavily focusing on available test scores, a 
concern presumably put to rest by his new 
findings regarding college persistence and 
completion. Finally, Harris acknowledges 
that the reforms were coupled with “a 
massive influx of federal and philanthropic 
funding and skilled people from across the 
country,” which may be difficult to replicate 
elsewhere. 


Harris concludes that: “Other districts 
should look to New Orleans, but tread 
carefully.”3 More recently, Harris explained 
that “In some sense, any policy advocacy 
based on research requires some degree 
of extrapolation.”4 


While | concur with Harrris’s critique, | 

go further in suggesting that the policy 
context of New Orleans - the dramatic 
disruption of low-income housing 

stock and a substantial reduction of 
concentrated poverty induced by Katrina 
make it very difficult, if not entirely 
implausible, to extrapolate NOLA findings 
to any other policy context. Further, in 
many policy contexts there the assumption 


that providing “choice” replaces the need to 
provide additional resources. That is, it is 
assumed that regardless of funding levels, 
where choice is provided, some excellent 
choices will emerge, and those choices will 
be the ones to survive market pressures, 
resulting in an improved system. This 
political tendency to offer choice and 
market-based reforms as a substitute, 
rather than as a complement for additional 
financial investment, makes it even less 
likely that other school systems would 
realize the benefits of similar structural 
changes.5 


RESEARCH DESIGN, EMPIRICAL 
EVIDENCE & POLICY 
IMPLICATIONS 


Here, | offer a model of how Harris and 
Larsen characterize their methods and 
findings. As explained above, Harris 

and Larsen consider the “package of 
reforms” or “market-based reforms” to 

the “treatment” they studied and thus 

the “cause” of the subsequent changes in 
student outcomes. As noted above, the 
authors make some attempts, given the 
available data, to address intervening 
factors, including children from low-income 
households (FRPL - Free or Reduced-priced 
Lunch), students with disabilities (SWD), 
English language learners (ELL) and racial 
groups. 


(Over-Simplified) Model of New Orleans School Reform Effects 
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Figure 1 


However, if we want to ascertain how 

to replicate elsewhere the effects of the 
treatments studied in New Orleans, we 
would have to have a better idea as to 

what those treatments entail and a more 
comprehensive understanding of the policy 
context. Figure 2 makes an attempt at 
parsing a) the elements of the reforms, and 
b) additional contextual factors. For example, 
some of the elements of the reforms are the 
adoption of a citywide school choice system, 
managed via a centralized enrollment 
system. Whether or to what extent that 
alone can be considered a specific treatment 
is still questionable. In addition, the choice 
model was coupled with shifting governance 
to a “portfolio” system of schools, which is 
also ill-defined as a treatment. 


In fact, this “treatment” per se actually leads 
to multiple widely varied treatments in the 
form of different school managers operating 
different school models, providing varied 


Short Run Outcomes Medium Run Outcome: 


instructional strategies, using different 
discipline/behavior management policies 
and so on. In response to recent negative 
findings on outcomes from Texas charter 
schools, Harris himself explains: 


One clear pattern in the research is that “no 
excuses” schools seem to have more positive 
effects on typical student outcome measures 
than other kinds of charter schools. This is 
true in Boston as well as in the Dobbie and 
Fryer study. (Actually, the pattern with no 
excuses also aligns with the old effective 
schools literature.) New Orleans, too, has 
had a large share of schools that might be 
described as no excuses. 


No excuses schools also tend to spend 
more money, and we do see higher 
spending in New Orleans. It may be the 
combination of schooling model and 
spending.® 


As such, if the mix of providers and their 
access to resources in any other setting were 


substantially different than in NOLA, the 
outcomes might be different as well. Each 
operator - each school model - is in effect its 
own unique treatment. Adopting a portfolio 
managed choice system is not. 


| have listed two other factors in addition to 
the core elements of the “reform” package in 
Figure 2. The first comes from Doug Harris's 
own acknowledgement that after the storm, 
talent converged on New Orleans to aid in 
the rebuilding of the city and its schools. 
Not only that, because much of that talent 
was young, that talent was inexpensive and 
even today, those who came immediately 
following the storm are not yet senior 
teachers or administrators. Those who 
stayed are approaching mid-career. 


Often overlooked, but mentioned in the 
recent technical report, is the role of 
additional funding which accompanied 
the reforms. That funding is considered by 
the authors to be part of the treatment, 


however, additional financial resources 
could also be considered a separate 
treatment and a significant cause of 
improvement. Arguably, if the funding 
increase had occurred independent 

of the market-based reforms, more of 
that funding might have been spent on 
classroom teachers, programs and services 
and less on transportation and duplicative 
administration which are expenses 
associated with “choice” systems. 


Finally in Figure 2, while Harris and Larsen 
do account for population changes, their 
measures are insufficient to capture one 

of the most substantial changes in NOLA 
over time. That is, the substantial reduction 
in concentrated poverty - concentrated 
geographically by neighborhoods and 
concentrated at the very lowest levels of 
family income. Harris and Larsen’s attempts 
to dismiss the inadequacy of their measures 
are also insufficient. 


Model of New Orleans School Reform Effects: Parsing the Left Hand Side of the Mode! 
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Harris and Larsen’s technical report 
identifies the two major “threats to 
validity” of their claim that “market based 
reforms” specifically were the cause of 
Substantive changes in student outcomes.’ 
However, as | dissect in the following two 
sections, Harris and Larsen’s dismissal of 
these validity threats is unsatisfactory. 

It is equally if not more reasonable to 
conclude that a) the substantive infusion 
of resources, regardless of accompanying 
reforms coupled with b) the dramatic 
reduction in concentrated poverty, were 
the main causes of improved measured 
student outcomes post-Katrina. 


FISCAL & RESOURCE TRENDS 


Here, | explore the question of changes to 
financial resources in New Orleans schools 
following the storm. Fiscal resource trends 
are reported in a series of figures in 
Appendix A. In conjunction with per pupil 
Spending increases, numbers of staff per 
pupil also increased substantially (largely 
due to reductions in the numbers of pupils) 
(Figure A7). We can expect significant 
benefits of reduced pupil to teacher ratios 
resulting in reduced class sizes. 


Related reports by Doug Harris have 
outlined resource changes in significant 
detail. Harris explains that “New Orleans’ 
publicly funded schools spent 13% ($1,358 


per student) more per pupil on operating 
expenditures than the comparison group 
after the reforms, even though the 
comparison group had nearly identical 
spending before the reforms.” Further, 
Spending on administration increased 
substantially (66 percent, or nearly 

$700 per pupil) relative to other similar 
schools statewide, more than half of 
which was attributable to administrative 
salaries. Instructional expenditures per 
pupil declined by a margin similar to the 
increase in administrative spending. About 
half of that decline was related to reduced 
Staff benefits, with the next largest decline 
being in staffing salaries, consistent with 
prior studies that show that charter school 
Staff is less-experienced and lower-paid.® 
Finally, Harris notes that “transportation 
Spending and other expenditures, which 
typically include contracts to outside firms, 
each increased by 33%."9 


To summarize, spending increased 
substantially, but a significant share of 
that spending was consumed by increased 
transportation expenses as well as higher 
administrative and overhead expense. 
Instructional staffing expenses were 

held artificially low due to the influx of a 
relatively inexperienced teacher workforce, 
and changes to pensions and other 
benefits. It is likely that these expense 
reductions are not sustainable over time, 
meaning that total spending will either 
have to increase further to maintain the 


system, or that other expenses will need to 
be substantially reduced. 


Harris and Larsen acknowledge in their 
technical report that such an infusion 

of spending might in fact “explain a 
substantial share of our estimated 
effects.” (p. 41) They draw this conclusion 
by considering the effects of spending 
increases estimated in a handful of recent 
major national studies, including the work 
of Jackson, Johnson, & Persico (2016) and 
similar work by Lafortune, Rothstein, 

and Schanzenbach (2016).1° But then, 
Harris and Larsen apply contorted logic 
to discount the possibility that increased 
spending of this magnitude alone might 


lead to similarly positive student outcomes. 


They base this argument on the following 
assumptions: 


1. That “the corruption and dysfunction in 
the Orleans Parish School Board prior 
to the storm implies that the additional 
resources would not have been used 
to generate better outcomes to the 
extent that the average district did in 
the above school funding studies.” (p. 
42) and 

2. “the city's spending increase, which 
came mainly from local funding and 
philanthropists, may have been partly 
caused by the reforms,” and therefore 
“Any effect of spending on student 
outcomes, in this sense, may not 
be just an alternative explanation, 


but rather an indirect effect of the 
reforms.” (p. 42) 


On the first point, the first mention 

of corruption induced inefficiency in 
Orleans Parish before the storm occurs 
in this explanation, without any empirical 
validation. That is, that the supposed 
corruption in the district necessarily 

led to a reduction in the efficiency with 
which the district produced student 
outcomes, relative to other districts. 
Thus, this assertion is purely speculative. 
Harris reiterates this assertion in a recent 
interview, claiming: “Pouring money into 
a failing district isn’t the answer, nor is it 
politically plausible in the long run.” 


The studies by Jackson, Johnson and 
Persico, as well as Lafortune, Rothstein, 
and Schanzenbach address the infusion 

of additional spending to schools and 
districts, without judgement as to whether 
any, most or all of the recipient districts of 
additional funding were previously “failing,” 
finding that the infusion of funding alone 
yielded substantive positive effects. 


Figure A8 (Appendix A) shows that prior to 
Katrina, Orleans Parish spending remained 
below the average for the metro area, and 
poverty was well above average. That is, no 
one ever attempted a substantive infusion 
of funding into the old New Orleans school 
district to see if it might have a positive 
effect. 


The second argument is perhaps even 
Stranger. The authors argue that the 
money would not have been there but for 
the reforms, thus it is unreasonable to 
consider what might have occurred if the 
money were there, absent the reforms. 
They argue that because the reforms 
attracted the philanthropic money, which 
helped result in better outcomes, the 
reforms are the primary causal agent. 


This argument does not at all negate 

the probability that similar investment 
without similar reforms may indeed result 
in similar outcome changes, given the 
demographic changes which also occurred 
during the period. Further, as noted 
above, the infusion of money coupled 

with the citywide choice and decentralized 
management resulted in a sizeable portion 
of the increased spending being diverted 
to transportation services and higher 
overhead expenses. Had the funding 
infusion occurred without the reforms - 
the true counterfactual - a larger share 
may have been directed toward classrooms 
and instructional staff. Arguably, similar 
spending increases allocated toward 
human resources in direct contact with 
students, rather than being dispersed 
through the market-based reform model, 
may have had even larger positive effects 
(similar in magnitude to those estimated by 
Jackson, Johnson and Persico, 2016).12 


DEMOGRAPHIC TRENDS 


In a recent interview, Doug Harris 
continues to assert that “The New Orleans 
population was disadvantaged before 

and that stayed about the same.”"3 When 
considering average shares of children 
who qualify for free or reduced priced 
lunch (under 185% income threshold for 
poverty), or other measures of central 
tendency (means, medians) for the city as 
a whole, this may appear true. But, there 
have indeed been substantial changes in 
the distribution of poverty across schools 
and neighborhoods and the concentration 
of extreme poverty in New Orleans. 


A 2015 report from the Brookings 
Institution found: 


Our analysis indicates, however, that the 
Share of the city’s poor residents living 

in neighborhoods of extreme poverty 
dropped from 39 percent in 2000 to 30 
percent in 2009-13 (the latest small-area 
data available). This drop occurred at the 
same time that concentrated poverty rose 
dramatically in many major American 
cities, spurred by the Great Recession and 
Slow recovery. As a result, whereas New 
Orleans ranked second among big U.S. 
cities in concentrated poverty prior to the 
storm, it ranked just 40th by 2009-13 (see 
Appendix table). 


Table 1 compares the change in 
concentrated poverty in New Orleans to 
that of selected other major cities with 
large shares of children attending charter 
schools. While Detroit, for example, 
experienced a 48% increase in child 
poverty, New Orleans experienced a 9% 
decrease. 


Others have characterized similar changes 
in extreme poverty and economically 
imbalanced neighborhood revitalization in 
New Orleans. ' 


2019 - 2013 


Concentrated 


Change 


New Orleans, 


poverty rate 


Harris and Larsen insist throughout their 
technical report that they have sufficiently 
tackled this potential threat to validity of 
their findings. Specifically, their method 
accounts for whether or not children are 
from families that fall below the 185% 
income threshold for poverty, as well as 
race. But their method misses entirely two 
important factors: 


1. The geographic concentration of 
poverty by neighborhoods, and 


2. The gradients of child poverty/family 


2000 


Concentrated 


Rank Poverty Rate eT al 


_ -9% 30% 40 39% 2 
Louisiana 
Newark, New 
1% 32% 32 32% 10 
Jersey 
Washington, 
District of 3% 24% 58 21% 38 
Columbia 
Philadelphia, 
2 11% 37% 26 27% 21 
Pennsylvania 
Memphis, 
16% 39% 22 23% 34 
Tennessee 
Kansas City, 
Missouri 24% 31% 39 7% 82 
Detroit, Michigan 48% 63% 3 16% 45 


Source: Brookings Institution analysis of GeoLytics Neighborhood Change 


Database and American Community Survey data 


income below the 185% income 
threshold for poverty. 


Harris and Larsen’s Footnote #13 partially 
addresses the critique that their findings 
may not fully account for the change 

in (average improvement of) housing 
stock after the storm, for those who 
returned." But this footnote, along with 
later references to public housing, are 
insufficient to dismiss the Brookings 
findings of substantial reduction to 
concentrated poverty. 


The authors present a handful of other 
comparisons in their attempt dismiss this 
threat to the validity of their finding that 
“reforms” not demographics (and spatial 
distribution) were the primary cause of 
outcome changes. 


The authors report 3 analyses in Table 2 of 
the technical report: 


1. Comparison of the pre-treatment 
scores for 3rd graders in NOLA versus 
other hurricane affected districts, 
showing that “returners” to NOLA had 
lower pre-treatment scores (Table 2); 


2. Comparison of changes to median 
family income, child poverty and adult 
education levels between NOLA and 
other hurricane affected districts, 
revealing only small differences; 


3. Tests of the relationship (and 
predictive validity) between child 
poverty, parent education measures 
and student outcomes at grade 3, 5 
and 8 using a national longitudinal 
database. 


Given these supplemental analyses, Harris 
concludes that demographic change is 

an insignificant concern, noting “We're 
confident about that because we come to 
the same conclusion from three entirely 
different types of analysis.” 


The first comparison of pre-treatment 
scores of 3rd graders is of limited value, 
focusing on differences between returners 
in NOLA and comparison districts, and 

not capturing to any extent the economic 
and neighborhood conditions under which 
these children live. Notably, economic 
background and neighborhood conditions 
affect not only a child’s starting point 
(where grade 3 is well beyond that starting 
point) but also the rate at which children 
progress over time. 


The second comparison, focusing on 
median family income, shares of adults 
with a BA or higher, or HS or lower, 

and children in poverty (100% income 
threshold) also fails to capture important 
gradients in economic status, as well as 
geographic concentration. The Brookings 
report also finds that overall poverty rates 
remained relatively unchanged, despite large 


reductions in concentrated poverty." 


The third analysis drawing on the Early 
Childhood Longitudinal Study (ECLS), while 
a clever attempt to discern the extent 

to which marginally richer data might 

lead to more accurate prediction also 
remains insufficiently precise to capture 
underlying demographic changes. Yes, 

the data are richer and more precise, 

but still not rich enough to capture the 
substantive differences in pre and post- 
Katrina populations or their geographic/ 
neighborhood concentration. The authors 
find that even this marginal improvement to 
precision suggests that their own estimates 
(of the reform effects) might be upwardly 
biased.'? 


These are all reasonable attempts to clarify 
the extent of potential bias in their models, 
which make use of the best available 

data. However, the following conclusions 


expressed by the authors are over-confident: 


> “Overall, it appears that the 
elimination of public housing and the 
disproportionate impact of flooding 
on low-income neighborhoods had 
a minimal effect on the relative 
demographics or test scores of the 
public school population in the years 
after the hurricanes.” (p. 25) and, 


- “This evidence suggests that 
population change is not a major 


threat to identification in the pooled 
analysis, especially after controlling for 
measurable demographic changes.” (p. 
25) 


The Brookings findings cited above reveal 
substantial reduction of concentrated 
poverty, whether directly as a function of 
flooding and changes to public housing, or 
due to other factors. That population change 
cannot be ignored and is not resolved by 
the comparisons provided by Harris and 
Larsen. The effects of child poverty and 
more specifically spatially concentrated child 
poverty, intergenerational poverty, and the 
duration of poverty exposure, all matter 
greatly when it comes to short and longer- 
term outcomes.” 


Indeed, one of the most significant factors 
affecting both the level and ongoing 
trajectory of student outcomes is child 
poverty - as it affects both individuals and 
groups of children concentrated by their 
neighborhoods, schools and classrooms. 
Both individual and concentrated poverty 
greatly affect children’s outcomes. As such, 
disrupting concentrated urban poverty may 
be one of the most effective possible reform 
Strategies available.”" 


The following two graphs illustrate the 
distribution of child poverty before and after 
the storm for public school enrolled children 
between the ages of 5 and 17 residing 

in New Orleans city and in New Orleans 


metropolitan area. Figure 3, for example, 
shows the distribution of the poverty index 
for families with children in New Orleans 
from Census 2000 and from the American 
Community Survey for years 2012 through 
2016. The poverty index is set to 100 for the 
income threshold for poverty. Thus, the free 
lunch cut-point would be at a poverty index 
of 130 and reduced lunch at 185. Severe or 
extreme poverty would be at an index value 
of 50. Gray bars indicate the pre-Katrina 
period and transparent bars indicate the 
post-Katrina period. Figure 3 shows that the 
largest shares of children are from families 


not merely below, but well below the free or 
reduced price lunch thresholds. However, 
pre and post Katrina levels are substantially 
different. Notably, there are gray bars (pre- 
Kartina) that spike (high concentrations) 
well below the poverty threshold, but there 
are no similarly high transparent bars 
(post-Katrina), even though there are 

still generally higher concentrations of 
children below the 100% income threshold 
for povertyUsing National School Lunch 
Program qualifications as cut points to 
declare children as poor or non-poor 

is relatively imprecise as it declares all 
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children attending schools which have 
greater than 90% low-income children 
initially plummeted quite significantly, but 
have since rebounded. 


But Figure 6 shows the seemingly 
incongruous finding that shares of children 
in schools where less than half of children 
are from low-income households have also 
increased in recent years. That is, while 
enrollments in very high poverty schools 


have increased, so too have enrollments 
in lower poverty schools. The citywide 
system is becoming more economically 
segregated, similar to the patterns found 
prior to Katrina. 


Figure 7 shows the shares of children 
attending schools that are predominantly 
(>90%) black versus the shares of children 
attending schools that are majority white. 
Although proportions of children in 
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those to the left of the 185 poverty index 
to be similarly poor, thus providing no 
information regarding the proportion of 
students who live in extreme poverty. 
The change is similar for the broader 
metropolitan area in Figure 4. Across 
the metropolitan area, the proportion of 
children in families at less than half the 
income threshold for poverty dropped 
from 19% to 14%. 


Figure 5, Figure 6 and Figure 7 display 
additional changes in the shares of 


children attending racially isolated and 
low-income isolated schools before and 
after Katrina. These figures are based on 
school enrollment data from the National 
Center for Education Statistics, Common 
Core of Data, Public School Universe file, 
for all schools within the city limits of 
New Orleans. Importantly, these figures 
show that not only was there a significant 
disruption to enrollment patterns at the 
point of Katrina, but that there are also 
continuing changes in enrollments. Figure 
5, for example shows that the share of 
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schools that are predominantly black are 
declining, proportions of children attending 
schools that are predominantly white are 
increasing and already at much higher 
levels than they were before the storm. 


Peer concentration and the geographic 
distribution of poverty matter. And 
changes to peer composition cut both 
ways - learning in the presence of high 
performing peers can have positive effects, 
and vice versa. Katrina provided unique 
opportunities to studies these effects in 
both directions, as children from low- 
income New Orleans families were largely 
displaced to Houston area schools. 


Imberman, Kugler and Sarcedote (2012) 
explored the peer effects of children 
displaced form NOLA and other schools 

on students in Houston schools which 
received many of these displaced students. 
Imberman and colleagues found that 
“student achievement improves with high 
achieving peers and worsens with low 
achieving peers. Finally, an increase in 

the inflow of evacuees raised incumbent 
absenteeism and disciplinary problems in 
Houston's secondary schools. (p. 2048)?2 
One might expect similar effects for those 
returning to NOLA schools in the years that 
followed, where the most disadvantaged 
students and families never returned, 
yielding substantively different though not 
readily observable classroom composition 
than before the storm. 


CONCLUSIONS & IMPLICATIONS 


To summarize, the following factors 
likely contribute substantially to the 
estimated gains in measured outcomes 
during the post-Katrina era in New Orleans: 


1. Asignificant reduction in concentrated 
poverty. Individual and collective 
poverty are well understood to 
have substantial adverse effects on 
Short-term and longer term student 
outcomes, therefore reductions 
in concentrated poverty result in 
improved outcomes; 


2. Asignificant infusion of additional 
resources, where several recent studies 
have estimated sizeable effects of 
spending increases on short-term and 
longer term student outcomes. 


Doug Harris and Matt Larsen downplay 
problems with the imprecision of poverty 
measures used in their analysis citing 
incomplete and insufficient data to dismiss 
the potential influence of demographic 
changes. Concentrated poverty in NOLA 
declined substantially after Katrina - more 
so than most other cities across the nation 
- and that change no doubt substantially 
influenced post-Katrina student outcome 
levels and gains. 


Doug Harris and Matt Larsen acknowledge 
that similar spending increases could 


explain a sizeable portion of the measured 
outcome gains, but then provide 
unsatisfactory explanations for why similar 
spending increases without the package 

of reforms either wouldn't have worked, 

or wouldn't have happened. The latter is 

a particularly weak argument. Harris and 
Larsen offer no empirical evidence that 
Orleans Parish schools were in fact less 
efficient than other districts prior to the 
storm. In fact, related reports do show 
that much of the increase in expenditures 
- because of the reform model - was 
diverted toward transportation and 
administrative expenses (redundant 
administrative structures between the 
coordinating Recovery District and multiple 
private managers). 


In education policy research the goal is to 
identify and precisely define a treatment or 
package of treatments and to evaluate the 
influence of those treatments on various 
outcome measures. Short run academic 
achievement outcomes, and longer term 
college attendance, persistence and 
completion rates are certainly important 
goals of education systems, and we should 
seek to identify cost-effective strategies to 
improve those outcomes. Unfortunately, 
New Orleans “reforms” are a difficult 
treatment to define. Harris and Larsen 
insist that funding increases are merely 
part of (if not a result of) the reforms and 
thus inseparable from other elements of 
the “reform” model. This is simply not true. 


While there are some identifiable elements 
of the “reforms” including citywide choice 
managed through a centralized enrollment 
system, the reforms essentially provide for 
choices among varied treatments, not any 
single treatment. 


Simply adopting a citywide choice and 
enrollment management system in other 
settings would by no means guarantee 
emergence of a similar array of treatments, 
distribution of providers, nor would 

it ensure that children are similarly 
distributed among the more effective 
treatments or providers. The enrollment 
management system is not the treatment. 
The schools, their programs and their 
teachers are. If we accept that some of 
the schools that entered the NOLA reform 
marketplace were the cause of improved 
student outcomes, above and beyond 
resource increases and demographic 
changes, then it is those schools, their 
specific models, strategies, teachers and 
leaders which are deserving of closer 
consideration for scaling up. 


Finally, public policy determinations, while 
considering treatment costs and commonly 
measured student outcome benefits, must 
also consider a broader array of questions 
regarding the public good and public 
interests. Some of these questions do not 
have easily quantifiable answers. 


We must, consider, for example, whether a 


system based on sorting students through 
individual and family choices can ever be 


an equitable and efficient system. Are some 


students still being left out? Falling through 
the cracks? And if so, who? 


Can we ensure that providers 

are available to meet all students 
needs and can we establish a 
funding model which accounts for 
differences in the distribution of 
students and their needs??3 Can 

we resolve structural inefficiencies 
caused by the continuous flow of 
students across institutions, including 
efficiently allocating capital assets 
(school buildings, classrooms) while 
maintaining public stewardship over 
those assets? 


Can we ensure the equal protection 
of student, employee, parent and 
taxpayer rights, or will some be 
asked to forgo legal protections as 

a tradeoff for the promise of a few 
additional test score points? Who 

is most likely to be asked to forgo 
those protections (children from low- 
income households, English language 
learners, children with disabilities)? 


Finally, we must consider the quality 
of life concerns caused by increased 
student travel times, more increased 
busing, the absence of a community 
school, and fewer walkable 


neighborhoods that result from choice 
systems.” 


To the extent possible, the goal in public 
policy research is to determine whether 

it might be reasonable to try similar 
treatments in other settings, expand or 
scale-up the treatment. Doing so requires 
thorough consideration of policy context 
and confounding conditions. In summary, 
by many measures, things look better 

now in New Orleans than they did before 
the storm. But those appearances are not 
necessarily caused by structural reforms/ 
market based strategies. New Orleans 

now is a new city, not comparable with 
New Orleans then. This new New Orleans 
does have a differently structured school 
system than before, but also has much less 
concentrated poverty and more resources 
in their schools. New Orleans is a novel city 
to begin with, but what happened to that 
city as aresult of Hurricane Katrina, and 

all that followed, creates a context which is 
entirely unique and incomparable to other 
cities and settings across the country, thus 
severely limiting any policy implications 
which can be drawn from studies of New 
Orleans school reforms. 
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APPENDIX A: RESOURCE CHANGES 
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https://www.brookings.edu/blog/the-avenue/2015/08/27/concentrated-poverty-in-new-orle- 
ans-10-years-after-katrina/ 
https://www.washingtonpost.com/posteverything/wp/2015/08/31/how-everyone-is-getting-it-wrong-on- 
new-orleans-school-reform/?utm_term=.340459b7f57a 
https://www.washingtonpost.com/posteverything/wp/2015/08/31/how-everyone-is-getting-it-wrong-on- 
new-orleans-school-reform/?utm_term=.340459b7f57a 
https://nepc.colorado.edu/publication/newsletter- NOLA-081418 

See, for example: https://schoolfinance101.wordpress.com/2017/10/09/choice-as-a-substitute-for-ade- 
quacy/ 

https://nepc.colorado.edu/publication/newsletter- NOLA-081418 

Harris and Larsen assert: “With additional analysis, we are able to largely rule out several threats to 
identification, including population change, strategic behavior from performance-based accountability, 
trauma and disruption from the hurricane, and the effectiveness of the interim schools that evacuated 
students temporarily attended. The treatment effects appear to be an order of magnitude larger than the 
potential biases, and some biases partially cancel out. The main potential alternative explanation for the 
effects, beyond the reforms themselves, is that the charter-based system gained a considerable financial 
advantage over the comparison groups (Buerger & Harris, 2016), though these alone cannot explain the 
overall effects.” (p. 7) 

Dennis Epple, Richard Romano, and Ron Zimmer, “Charter Schools: A Survey of Research on Their 
Characteristics and Effectiveness,’ in Handbook of the Economics of Education, vol. 5 (Amsterdam: Else- 
vier, 2016), 139-208. 

Christian Buerger and Douglas N. Harris, How Did the New Orleans School Reforms Influence School 
Spending? (technical report, Education Research Alliance for New Orleans, New Orleans, 2017), https:// 
educationresearchalliancenola.org/publications/does-school-reform-spending-reform-the-effect-of-the- 
new-orleans-school-reforms-on-the-use-and-level-of-school-expenditures. 


. I provide a thorough discussion of these and related studies at: Baker, B. D. (2016). Does money matter in 


education?. Albert Shanker Institute. https://files.eric.ed.gov/fulltext/ED563793.pdf 


. https://nepc.colorado.edu/publication/newsletter- NOLA-081418 
. For a thorough discussion, see Baker, B. D. (2017). How Money Matters for Schools. https://learningpoli- 


cyinstitute.org/sites/default/files/product-files/How_Money_Matters_REPORT.pdf 


. https://nepc.colorado.edu/publication/newsletter- NOLA-081418 
. https://www.brookings.edu/blog/the-avenue/2015/08/27/concentrated-poverty-in-new-orle- 


ans-10-years-after-katrina/ 


. Rivlin, G. (2016) White New Orleans Has Recovered from Hurricane Katrina. Black New Orleans Has 


Not. https://talkpoverty.org/2016/08/29/white-new-orleans-recovered-hurricane-katrina-black-new-or- 
leans-not/ New York Times. Ten Years After Katrina. https://www.nytimes.com/interactive/2015/08/26/ 
us/ten-years-after-katrina.html Fussell, E., Sastry, N., & VanLandingham, M. (2010). Race, socioeconom- 
ic status, and return migration to New Orleans after Hurricane Katrina. Population and environment, 
31(1-3), 20-42.Bliss, L. (2015) 10 Years Later, There's So Much We Don't Know About Where Katrina 
Survivors Ended Up https://www.citylab.com/equity/2015/08/10-years-later-theres-still-a-lot-we-dont- 
know-about-where-katrina-survivors-ended-up/401216/ 

A related possibility is that the students who evacuated benefited from being in better housing and/or 
neighborhoods. It is possible that, as in some past disasters, New Orleans’ housing quickly surpassed 
the quality that had preceded it and that this drove positive educational effects for those who did return 
(Hornbeck & Keniston, 2017). This seems unlikely, however, as the quality of the New Orleans’ hous- 
ing stock was unchanged for non-flooded homes for sale and remained considerably worse for flooded 
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homes one full year after Katrina (McKenzie & Levendis, 2010). Most homeowners lacked flood insur- 
ance, and state and federal programs averaged about $40,000 per home, or about 17 percent of their prior 
values. This likely understates the percentage of the housing value covered for low-income homeowners, 
but not enough to generate an overall improvement in the housing stock for low-income families. It 

is also worth considering whether there was a general improvement in neighborhood quality after the 
storms, in which case the effects might be interpreted more in line with the well known Moving to Op- 
portunity (MTO) program (Ludwig et al., 2013). However, this, and other evidence presented later about 
poverty, suggests that the improvements in student outcomes we observe cannot plausibly be explained 
by neighborhood effects either. 

https://nepc.colorado.edu/publication/newsletter- NOLA-081418 

Noting: “As the Data Center observes in its recent post-Katrina look at the region, the poverty rate in the 
city of New Orleans in 2013 (27 percent) was statistically unchanged from 2000” https://www.brookings. 
edu/blog/the-avenue/2015/08/27/concentrated-poverty-in-new-orleans- 10-years-after-katrina/ 

Noting: “The latter number suggests a possible upward bias in the treatment effects estimates in the 
pooled analysis, but all of the estimates are small in magnitude relative to the average treatment effects we 
show later.’ (p. 25) 

Coley, R. J., & Baker, B. (2013). Poverty and education: Finding the way forward. Educational Testing 
Service Center for Research on Human Capital and Education. Michelmore, K., & Dynarski, S. (2016). 
The gap within the gap: Using longitudinal data to understand income differences in student achieve- 
ment (No. w22474). National Bureau of Economic Research. Lubienski, S., & Crane, C. C. (2010). Be- 
yond free lunch: Which family background measures matter?. education policy analysis archives, 18, 11. 
Duncombe, W., & Yinger, J. (2005). How much more does a disadvantaged student cost?. Economics of 
Education Review, 24(5), 513-532. Orfield, G., & Lee, C. (2005). Why segregation matters: Poverty and 
educational inequality. Civil Rights Project at Harvard University (The). Duncan, G. J., & Murnane, R. J. 
(Eds.). (2011). Whither opportunity?: Rising inequality, schools, and children's life chances. Russell Sage 
Foundation. 

Chetty, R., Hendren, N., & Katz, L. F. (2016). The effects of exposure to better neighborhoods on chil- 
dren: New evidence from the Moving to Opportunity experiment. American Economic Review, 106(4), 
855-902. 

Imberman, S. A., Kugler, A. D., & Sacerdote, B. I. (2012). Katrina's children: Evidence on the structure of 
peer effects from hurricane evacuees. American Economic Review, 102(5), 2048-82. 

Anecdotally, one former NOLA Charter School board member explains: The second problem, and one 
of the most fatal to Cypress, is the clash of a competitive system with a fixed-price economy. Cypress 
wanted to educate all children, and twenty-six per cent of its students had special needs, twice the city’s 
average. New Orleans’s funding for these students, though much improved, is still a work in progress. 
Each special-needs student effectively takes money away from the budget for other students. Given this, 
other schools “recommended” Cypress to parents of special-needs families. Cypress welcomed those chil- 
dren; it was its mission. But unlike, say, a startup restaurant chain that could adjust its menu or prices to 
attract certain customers and improve margins, Cypress is an open-admission school. It couldn't, morally 
or legally, choose its customers nor the revenue it could gain from each one. https://www.newyorker. 
com/business/currency/what-new-orleans-tells-us-about-the-perils-of-putting-schools-on-the-free-mar- 
ket See also: Baker, B. D., Libby, K., & Wiley, K. (2015). Charter School Expansion and Within-District 
Equity: Confluence or Conflict?. Education Finance and Policy, 10(3), 423-465. 

Baker, B. (2016). Exploring the consequences of charter school expansion in US cities. Economic Policy 
Institute, November, 30. 
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